Image processing apparatus and method

ABSTRACT

The invention provides an image processing apparatus and method wherein image signals can be arithmetically processed on the real-time basis. Infrared rays introduced into a light reception section are photoelectrically converted in synchronism with a reset pulse signal supplied from a timing generator and are outputted to an amplification section in synchronism with a light reception section transfer pulse signal supplied from the timing generator. The signal inputted to the amplification section is amplified to a level necessary for processing in an apparatus in the following stage in synchronism with an amplification section drive pulse signal supplied thereto from the timing generator, and outputted to an arithmetic operation section. The signal inputted to the arithmetic operation section is temporarily stored into a storage section, and a predetermined arithmetic operation designated by an arithmetic operation selection signal from an arithmetic operation control section is performed for the signal by a comparison section to produce a binary digitized signal. The binary digitized signal is outputted to an outputting section. The signal inputted to the outputting section is outputted as a pixel signal over a common signal line in synchronism with a selection signal from a horizontal scanning circuit.

BACKGROUND OF THE INVENTION

This invention relates to an image processing apparatus and method.

A technique of arithmetically operating an image signal is being spreadwidely. This technique is utilized, for example, to determine athree-dimensional image of an imaging object by arithmetically operatingan image signal.

In order to obtain a plurality of signals necessary for arithmeticoperation, where-such an image pickup device as a CCD (Charge CoupledDevice) is used, an image of an imaging object is picked uprepetitively. Image signals obtained by the repetitive image picking upoperations are stored into a memory device such as a frame memory. Then,the thus stored signals are read out from the storage device and usedfor such arithmetic operation as described above.

Also such a non-scanning type image pickup device as disclosed inJapanese Patent Publication No. hei 6-25653 has been proposed as amethod and an apparatus for realizing real-time geometric measurement.

However, where an image of an imaging object is picked up repetitively,since the time of, for example, 33.3 msec or 16.6 msec is required for asingle image picking up operation, there is a subject to be solved thatthis time required for the image pickup makes an upper limit and aresult of arithmetic operation of image information cannot be obtainedat a higher rate.

Also it is a subject to be solved that, since a result of arithmeticoperation of an image signal cannot be obtained unless an image pickingup operation is performed repetitively, a result of arithmetic operationcannot be obtained on the real-time basis.

In the non-scanning type image pickup device disclosed in JapanesePatent Publication No. hei 6-25653, since outputs of individual pixelsarrayed on the image pickup device are handled independently of oneanother, output signal lines of the pixels cannot be formed as a commonoutput signal line or lines. Further, since the non-scanning type imagepickup device does not include storage means for the pixels, it has asubject to be solved that it loses such a characteristic of the“non-scanning type” that individual pixels operate independently of oneanother and consequently cannot perform real-time processing.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide an image processingapparatus and method wherein image signals obtained by photoelectricconversion of light received individually by elements for receivinglight from an object to be imaged can be arithmetically processed on thereal-time basis.

In order to attain the object described above, according to the presentinvention, image signals obtained by photoelectric conversion of lightreceived individually by elements for receiving light from an object tobe imaged are arithmetically operated in accordance with a predeterminedrule.

More particularly, according to an aspect of the present invention,there is provided an image processing apparatus having an optical areain which a plurality of elements are disposed in a matrix, comprisinglight reception means for receiving light introduced into the elementsof the optical area and photoelectrically converting the light,arithmetic operation means for arithmetically operating a signalobtained for each of the elements by the photoelectric conversion by thelight reception means in accordance with a predetermined rule,outputting means for outputting a result of the arithmetic operation ofthe arithmetic operation means for each of the elements, and timingadjustment means for adjusting a timing at which the result of thearithmetic operation is to be outputted for each of the plurality ofelements from the outputting means.

The arithmetic operation means may include storage means forsuccessively storing a plurality of signals at different timingsobtained by the photoelectric conversion. In this instance, thearithmetic operation means may execute comparison arithmetic operationfor a combination of a plurality of ones of the signals stored in thestorage means. The comparison arithmetic operation may include anarithmetic operation for determining a maximum value or a minimum valueof the signal.

The outputting means may output results of the arithmetic operation foreach of the rows or the columns of the elements at a timing adjusted bythe timing adjustment means.

According to another aspect of the present invention, there is providedan image processing method for an image processing apparatus which hasan optical area in which a plurality of elements are disposed in amatrix, comprising a light reception step of receiving light introducedinto the elements of the optical area and photoelectrically convertingthe light, an arithmetic operation step of arithmetically operating asignal obtained for each of the elements by the photoelectric conversionof the processing in the light reception step in accordance with apredetermined rule, an outputting step of outputting a result of thearithmetic operation of the processing in the arithmetic operation stepfor each of the elements, and a timing adjustment step of adjusting atiming at which the result of the arithmetic operation is to beoutputted for each of the plurality of elements by the processing in theoutputting step.

In the image processing apparatus and the image processing method, lightintroduced into each of the elements of the optical area isphotoelectrically converted, and a signal obtained by the photoelectricconversion for each of the elements is arithmetically operated inaccordance with the predetermined rule. Then, a result of the arithmeticoperation is outputted for each of the elements. Consequently,arithmetic operation processing of image information can be performed onthe real-time basis.

The above and other objects, features and advantages of the presentinvention will become apparent from the following description and theappended claims, taken in conjunction with the accompanying drawings inwhich like parts or elements denoted by like reference symbols.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a construction of an image processingapparatus to which the present invention is applied;

FIG. 2 is a block diagram showing a construction of a distance sensor ofthe horizontal type shown in FIG. 1;

FIG. 3 is a block diagram showing a construction of a distance sensor ofthe vertical type shown in FIG. 1;

FIG. 4 is a block diagram showing a detailed construction of pictureelements shown in FIG. 2;

FIG. 5 is a graph illustrating an arithmetic operation method of a peakof light received;

FIG. 6 is a diagrammatic view showing a detailed construction of anarithmetic operation section shown in FIG. 4;

FIG. 7 is a flow chart illustrating operation of the image processingapparatus shown in FIG. 1;

FIG. 8 is a flow chart illustrating operation of a picture element shownin FIG. 2;

FIG. 9 is a timing chart illustrating operation of a picture elementshown in FIG. 2;

FIG. 10 is a block diagram showing another construction of a distancesensor shown in FIG. 1; and

FIG. 11 is a timing chart illustrating operation of the distance sensorof FIG. 10.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring first to FIG. 1, there is shown an image processing apparatusto which the present invention is applied. The image processingapparatus shown is generally denoted at 1 and includes a pattern lightprojection section 12, an imager 15, a video signal processing section16, a distance sensor 17, a shape data processing section 18, and asystem control section 11 which controls operation of the elements justmentioned.

The pattern light projection section 12 irradiates infrared rays of apattern necessary for distance measurement toward an imaging object 2 inaccordance with an instruction from the system control section 11. Forthe pattern light, slit light or grid light is used based on a principleof measurement of the distance sensor 17.

A lens 13 condenses light from the imaging object 2 and introduces thelight into a prism 14. The prism 14 spectrally separates the incidentlight from the lens 13 into visible rays and infrared rays. Inparticular, since light from the imaging object includes not onlyvisible rays but also reflected light from the imaging object of theinfrared rays irradiated from the pattern light projection section 12described above, it is spectrally separated into the visible rays andthe infrared rays and outputs the visible rays to the imager 15 and theinfrared rays to the distance sensor 17.

The imager 15 is formed from a CCD (Charge Coupled Device), a CMOS(Complementary Metal Oxide Semiconductor) or the like, and extractscolor information from the visible rays inputted thereto from the prism14 in response to a synchronizing signal, a control signal and so forthfrom the system control section 11 and outputs the extracted colorinformation as a video signal to the video signal processing section 16.

The video signal processing section 16 performs gain adjustment, coloradjustment processing and so forth for the video signal inputted theretofrom the imager 15 in response to a synchronizing signal and a controlsignal from the system control section 11, converts, when necessary, theresulting video signal into an analog signal or a digital signal andoutputs the analog or digital signal as a color video signal to acomputer 19.

The distance sensor 17 receives the infrared rays introduced theretofrom the prism 14, processes the received infrared rays in response to asynchronizing signal and a control signal from the system controlsection 11 into a binary digitized signal, and outputs the resultingsignal to the shape data processing section 18. It is to be noted thatdetails of the distance sensor 17 are hereinafter described.

The shape data processing section 18 determines, in response to asynchronizing signal and a control signal from the system controlsection 11, a timing at which the intensity of the infrared raysexhibits its peak from the binary digitized signal inputted thereto fromthe distance sensor 17, calculates a distance to the imaging object 2based on the principle of triangular surveying from the determinedintensity, and arithmetically operates a three-dimensional shape of theimaging object 2. Then, the shape data processing section 18 outputs aresult of the arithmetic operation as a shape data signal to thecomputer 19.

The computer 19 performs computer graphics processing for the colorvideo signal supplied thereto from the video signal processing section16 and the shape data signal supplied thereto from the shape dataprocessing section 18. The computer 19 outputs resulting data to amonitor 3, which may be formed from a CRT (Cathode Ray Tube), a LCD(Liquid Crystal Display) or the like, or to an external storageapparatus 4 so that the data may be stored into the external storageapparatus 4.

Now, details of the distance sensor 17 are described with reference toFIGS. 2 and 3. Distance sensors are roughly divided into two types, andthe distance sensor 17 may be of any of the two types. In particular,where a first one of the two types is used, pixels 41 arrayed in ahorizontal direction in an optical area 31 are successively scanned toextract output signals to be processed from the pixels 41. On the otherhand, where a second one of the two types is used, pixels 41 arrayed ina vertical direction in an optical area 31 are successively scanned toextract output signals to be processed from the pixels 41. FIG. 2 showsa construction of the distance sensor 17 of the former type while FIG. 3shows a construction of the distance sensor 17 of the latter type.

First, the distance sensor 17 of the horizontally scanning type isdescribed with reference to FIG. 2.

The optical area 31 includes a plurality of pixels 41 having anarithmetic operation function and disposed in a matrix of n×m (=quantityin the horizontal direction×quantity in the vertical direction). Each ofthe pixels 41 arithmetically operates a signal corresponding to anamount of received light in response to a reset pulse signal and a lightreception section transfer pulse signal outputted from a timinggenerator 32, and outputs a result of the arithmetic operation in ahorizontal direction to an outputting circuit 34 over a common signalline 42 based on a selection signal supplied thereto from a horizontalscanning circuit 33 a. It is to be noted that the pixels 41 arehereinafter described in detail.

The timing generator 32 supplies control pulse signal to the horizontalscanning circuit 33 a and the outputting circuit 34 and generates andoutputs an amplification section drive pulse signal, a reset pulsesignal and a light reception section transfer pulse signal to the pixels41 of the optical area 31 in accordance with a control signal from thesystem control section 11.

The horizontal scanning circuit 33 a generates and supplies a clearpulse signal, a storage section transfer pulse signal, a comparisonsection drive pulse signal and a selection signal to the pixels 41 ofthe optical area 31 in accordance with a control pulse signal suppliedthereto from the timing generator 32.

The outputting circuit 34 successively receives output signals of thepixels 41 of the optical area 31 over the common signal lines 42 insynchronism with a control pulse signal from the timing generator 32 andoutputs the received output signals to the shape data processing section18.

An arithmetic operation control section 35 supplies an arithmeticoperation selection signal for selecting (designating) an arithmeticoperation process to be executed by a matrix circuit 72 (FIG. 6) of astorage section 61 of an arithmetic operation section 53 of each of thepixels 41 in accordance with a control signal from the system controlsection 11. It is to be noted that the matrix circuit 72 of the storagesection 61 of the arithmetic operation section 53 is hereinafterdescribed in detail.

On the other hand, in the distance sensor 17 of the vertical scanningtype shown in FIG. 3, a vertical scanning circuit 33 b is provided inplace of the horizontal scanning circuit 33 a of the distance sensor 17of the horizontal scanning type shown in FIG. 2, and outputs of thepixels 41 driven by the vertical scanning circuit 33 b are successivelysupplied in a vertical direction to the outputting circuit 34 over thecommon signal lines 42. The other construction of the distance sensor 17of FIG. 3 is similar to that of the distance sensor 17 of FIG. 2.

Now, details of the pixels 41 are described with reference to FIG. 4. InFIG. 4, n pixels 41 a to 41 n connected to one of the common signallines 42 shown in FIG. 2 are shown. Here, while a construction only ofthe pixel 41 a is shown, also the other pixels 41 b to 41 n have asimilar construction. This similarly applies to the distance sensor 17of FIG. 3.

A light reception section 51 of the pixel 41 a is formed from a lightreception element such as, for example, a photodiode. The lightreception section 51 receives infrared rays inputted thereto from theprism 14, photoelectrically converts the received infrared rays inresponse to a reset pulse signal supplied thereto from the timinggenerator 32, and outputs a resulting signal to an amplification section52 in response to a light reception section transfer pulse signalsupplied thereto from the timing generator 32.

The amplification section 52 amplifies the signal inputted thereto fromthe light reception section 51 to a level necessary for processing by anapparatus in the following stage in synchronism with an amplificationsection drive pulse signal supplied thereto from the timing generator32, and outputs the signal of the amplified level to the arithmeticoperation section 53.

The arithmetic operation section 53 includes a storage section 61 and acomparison section 62, and performs a predetermined arithmetic operationdesignated by an arithmetic operation selection signal from thearithmetic operation control section 35 for a signal inputted theretofrom the amplification section 52 to produce a binary digitized signaland outputs the binary digitized signal to an outputting section 54. Itis to be noted that the storage section 61 and the comparison section 62are hereinafter described in detail.

The outputting section 54 outputs a signal inputted thereto from thearithmetic operation section 53 over the common signal line 42 as apixel signal in synchronism with a selection signal from the horizontalscanning circuit 33 a to the outputting circuit 34.

Before the storage section 61 and the comparison section 62 of thearithmetic operation section 53 are described, an arithmetic operationof a binary digitized signal of the arithmetic operation section 53 isdescribed.

A signal corresponding to the amount of received light by the lightreception section 51 is amplified by the amplification section 52 andinputted to the arithmetic operation section 53. It is assumed that thesample signal s(k) of the intensity of the infrared rays of the receivedlight varies together with the timing k of sampling as shown in FIG. 5.In this instance, each time the timing k varies, the sample signal ofthe infrared rays intensity is indicated as given below:

-   -   s(k−3), s(k−2), s(k−1), s(k), s(k+1), s(k+2), s(k+3), . . .        It is to be noted that k−1 has a value prior in time to k.

In this instance, as a function for detecting the time at which theintensity of the infrared rays exhibits its peak, such a function g(k)representing a displacement difference as given by the followingexpression (1) is considered:g(k)={s(k)+s(k−1)}−{s(k−2)+s(k−3)}  (1)

This function g(k) substantially corresponds to differentiation of thesample signal s(k). If it is assumed that the function g(k) assumes apositive higher value as the sample signal s(k) of the infrared raysintensity increases, then when g(k)>0, the sample signal s(k) of theinfrared rays intensity indicates an increase with respect to thevariation of the time, but on the contrary when g(k)<0, the intensity ofthe sample signal s(k) of the infrared rays intensity indicates adecrease with respect to the variation of the time.

Accordingly, the timing k at which the function g(k) varies from apositive value to a negative value is the time at which the samplingsignal of the infrared rays intensity exhibits its peak.

Thus, the time at which a peak of the sample signal of the infrared raysintensity is detected can be determined by a similar technique to thatdescribed above using a function f(k) indicated by the followingexpression (2) wherein a bias of a predetermined level is added to thefunction g(k) taking noise of the sample signal s(k) of the infraredrays intensity into consideration as seen from FIG. 5:f(k)={s(k)+s(k−1)}−{s(k−2)+s(k−3)}+BIAS  (2)

As seen from FIG. 5, the function f(k) varies in response to a variationof the infrared rays intensity s(k), and the timing k at an intersectingpoint at which the value of the function f(k) illustrated in FIG. 5varies from a positive value to a negative value across the zero levelindicates the timing at which the infrared rays intensity s(k) exhibitsits peak. It is to be noted that, in FIG. 5, the peak of the infraredrays intensity s(k) is at the sampling timing (k−2) displaced from thetiming k. However, since this displacement has a fixed value whichdepends uniquely upon the function f(k), an accurate timing k at whichthe sample signal s(k) of the infrared rays intensity exhibits its peakcan be determined by multiplying the timing calculated as above by afixed offset.

The arithmetic operation section 53 outputs a binary digitized signal,which assumes 0 when the value of the function f(k) given above is inthe positive or zero but assumes 1 when the value of the function f(k)is in the negative. The outputting circuit 34 outputs the receivedbinary digitized signal as an output signal to the shape data processingsection 18 in the following stage. The shape data processing section 18in the following state determines the peak of the infrared raysintensity from the timing of sampling at which the infrared raysintensity determined from the binary digitized signal exhibits its peak,and calculates the distance to the imaging object from the peak value ofthe infrared rays intensity in accordance with a principle similar tothat of the triangular surveying.

Subsequently, the storage section 61 and the comparison section 62 ofthe arithmetic operation section 53 are described with reference to FIG.6.

Storage cells 71 a to 71 d of the storage section 61 successively storea signal inputted from the amplification section 52 as a sample signalof the infrared rays intensity in response to clear pulse signals CLR1to CLR4 sent thereto in synchronism with a sampling synchronizing signalfrom the horizontal scanning circuit 33 a.

In particular, if it is assumed that, for example, at a certain timingk, a sample signal s(k) is stored in the storage cell 71 a, anothersample signal s(k−1) in the storage cell 71 b, a further sample signals(k−2) in the storage cell 71 c and a still further sample signal s(k−3)in the storage cell 71 d, then at the next timing k+1, a clear pulsesignal CLR4 is sent from the horizontal scanning circuit 33 a to thestorage cell 71 d which has the oldest signal stored therein so that thesignal s(k−3) which is the preceding sample signal is erased from thestorage cell 71 d in synchronism with the clear pulse signal CLR4.Immediately after then, a signal from the amplification section 52 isinputted in synchronism with a reception light section transfer pulsesignal sent thereto from the horizontal scanning circuit 33 a, andthereupon, a sample signal s(k+1) which is a new sample signal of theinfrared rays intensity is stored into the storage cell 71 d.Thereafter, each time the timing k for sampling varies, a new samplesignal of the infrared rays intensity is successively rewritten andstored into the storage cell which has the oldest sample signal of theinfrared rays intensity currently stored therein similarly.

The sample signals of the infrared rays intensity stored in the storagecells 71 a to 71 d are outputted in parallel to a matrix circuit 72.

The matrix circuit 72 controls on/off switching of switches 81 a to 84a, 81 b to 84 b, 81 c to 84 c and 81 d to 84 d in response to anarithmetic operation selection signal from the arithmetic operationcontrol section 35. In particular, if the signals outputted from thestorage cells 71 a to 71 d are represented by signals V1 to V4 and, at acertain timing k, for example, a sample signal s(k) is stored in thestorage cell 71 a, another sample signal s(k−1) in the storage cell 71b, a further sample signal s(k−2) in the storage cell 71 c and a stillfurther sample signal s(k−3) in the storage cell 71 d, then the functionf(k) to be arithmetically operated is represented by the followingexpression (3):f(k)={s(k)+s(k−1)}−{s(k−2)+s(k−3)}+BIAS  (3)

Consequently,f(k)=V 1 +V 2−V 3−V 4+BIAS  (4)

Then at the next sampling timing k+1, the sample signal s(k−3) of theinfrared rays intensity at the oldest timing k−3 stored in the storagecell 71 d is replaced by a sample signal s(k+1). Consequently, theexpression to be arithmetically operated is given by the followingexpression:f(k+1)={s(k+1)+s(k)}−{s(k−1)+s(k−2)}+BIAS  (5)

Consequently,f(k+1)=V 4+V 1−V 2−V 3+BIAS  (6)

Then, each time the sampling timing varies, arithmetic operations of thefollowing four different expressions are repeated:f(k)=V 1+V 2−V 3−V 4+BIAS  (7)f(k+1)=V 4+V 1−V 2−V 3+BIAS  (8)f(k+2)=V 3+V 4−V 1−V 2+BIAS  (9)f(k+3)=V 2+V 3−V 4−V 1+BIAS  (10)

Where the combination of additions and subtractions is successivelychanged at each sampling timing to execute an arithmetic operation inthis manner, stored signals themselves need not be transferred betweenstorage cells. Consequently, deterioration of a signal and so forthwhich arises upon transfer can be suppressed.

Here, the arithmetic operation modes of the expressions (7) to (10)given above are defined as modes A to D, respectively.

Referring back to FIG. 6, the matrix circuit 72 controls on/offswitching of the switches 81 a to 84 a, 81 b to 84 b, 81 c to 84 c and81 d to 84 d in response to the arithmetic operation mode of anarithmetic operation selection signal transmitted thereto from thearithmetic operation control section 35. For example, if, at the timingk, an arithmetic operation selection signal of the mode A is transmittedfrom the arithmetic operation control section 35 to the matrix circuit72, then the matrix circuit 72 switches the switches 81 a to 84 a on sothat the signal V1 stored in the storage cell 71 a and the signal V2stored in the storage cell 71 b are supplied to a positive input of adifferential amplification circuit 93 of the comparison section 62 andthe signal V3 stored in the storage cell 71 c and the signal V4 storedin the storage cell 71 d are supplied to a negative input of thedifferential amplification circuit 93 of the comparison section 62.

A load 91 a to the comparison section 62 is connected to the positiveinput of the differential amplification circuit 93 while another load 91b to the comparison section 62 is connected to the negative input of thedifferential amplification circuit 93, and the loads 91 a and 91 bconvert currents inputted thereto from the storage cells 71 a to 71 dinto voltages. A variable current source 92 generates bias current andsupplies the bias current to the positive input of the differentialamplification circuit 93 so that the bias current is added as BIAS inthe expressions (7) to (10) to one of signals of the storage cells 71 ato 71 d which is inputted to the positive input of the differentialamplification circuit 93. The differential amplification circuit 93arithmetically operates a difference between signals to the positiveinput and the negative input thereof.

For example, if an arithmetic operation selection signal of the mode Ais inputted from the arithmetic operation control section 35 to thematrix circuit 72 of the storage section 61, then the switches 81 a to84 a are switched on, and the signal V1 stored in the storage cell 71 aand the signal V2 stored in the storage cell 71 b are inputted to thepositive input of the differential amplification circuit 93 of thecomparison section 62. Further, the signal V3 stored in the storage cell71 c and the signal V4 stored in the storage cell 71 d are inputted tothe negative input of the differential amplification circuit 93 of thecomparison section 62. Accordingly, the differential amplificationcircuit 93 executes the arithmetic operation of the expression (7).

It is to be noted that, while the comparison section 62 uses thedifferential amplification circuit 93, it may otherwise use a choppertype comparison circuit instead.

Subsequently, operation of the image processing apparatus 1 is describedwith reference to a flow chart of FIG. 7.

In step S1, pattern light (infrared rays) is generated by the patternlight projection section 12 in accordance with a control signal from thesystem control section 11 and irradiated toward the imaging object 2.Then, infrared rays and visible rays reflected from the imaging object 2are condensed by the lens 13 and introduced into the prism 14.

In step S2, the incident light is spectrally separated into the visiblerays and the infrared rays by the prism 14. The visible rays thusspectrally separated are introduced into the imager 15 while theinfrared rays are introduced into the distance sensor 17.

In step S3, the imager 15 extracts color information from the visiblerays and outputs the color information to the video signal processingsection 16. The video signal processing section 16 performs gainadjustment and color signal processing for the color informationinputted thereto and outputs resulting information as a color videosignal to the computer 19. Meanwhile, the distance sensor 17 receivesthe infrared rays at the pixels 41 thereof and produces and outputs abinary digitized signal, from which a peak of the intensity of theinfrared rays can be detected, to the shape data processing section 18.It is to be noted that the processing of the pixels 41 of the distancesensor 17 is hereinafter described. The shape data processing section 18determines a sampling timing at which the infrared rays exhibit a peakfrom the binary digitized signal from the distance sensor 17,arithmetically operates the distance to the imaging object 2 inaccordance with the principle of triangular surveying from the infraredrays intensity at the sampling timing, and outputs the resultingdistance as a shape data signal to the computer 19.

In step S4, the computer 19 combines the color video signal and theshape data signal inputted thereto, performs computer graphicsprocessing for the combined signal and outputs a resulting signal to themonitor 3 or outputs it to the external storage apparatus 4 so that itis stored into the external storage apparatus 4, thereby ending theprocessing.

Subsequently, operation when V1=s(k−4), V2=s(k−1), V3=s(k−2) andV4=s(k−3) as sample signals of the infrared rays intensity are stored inthe storage cells 71 a to 71 d, respectively, of the pixels 41 of thedistance sensor 17 of FIG. 2 at the sampling timing k−1 in the timingchart of FIG. 9 is described with reference to a flow chart of FIG. 8and a timing chart of FIG. 9.

If a reset pulse signal transmitted from the timing generator 32 isinputted to the light reception section 51 immediately after thesampling timing k−1 in step S11, then the light reception section 51resets the reception light level and starts reception of the infraredrays newly (an accumulation phase in FIG. 9).

In step S12, the light reception section 51 photoelectrically convertsthe newly received infrared rays in synchronism with a light receptionsection transfer pulse signal (not shown) from the timing generator 32and outputs a resulting signal to the amplification section 52.

In step S13, the amplification section 52 amplifies the signal inputtedthereto from the light reception section 51 in synchronism with anamplification section drive pulse signal (not shown) from the timinggenerator 32 and outputs the amplified signal to the arithmeticoperation section 53.

In step S14, the arithmetic operation section 53 erases the signals(k−4) of the storage cell 71 a, which is the oldest signal, insynchronism with a clear pulse signal CLR1 (FIG. 9) from the horizontalscanning circuit 33 a, and then stores the signal from the amplificationsection 52 into the storage cell 71 a in synchronism with a storagesection transfer pulse signal TX1 (FIG. 9) from the horizontal scanningcircuit 33 a.

In step S15, the storage cells 71 a to 71 d outputs the signals V1 to V4stored therein to the matrix circuit 72.

In step S16, the matrix circuit 72 switches the switches 81 a to 84 a onin response to a signal of the mode A of an arithmetic operationselection signal from the arithmetic operation control section 35. Ofthe signals V1 to V4 inputted from the storage cells 71 a to 71 d, thesignals V1 and V2 are supplied to the positive input of the differentialamplification circuit 93 of the comparison section 62, and the signalsV3 and V4 are inputted to the negative input of the differentialamplification circuit 93.

In step S17, a signal obtained by adding the bias BIAS supplied from thevariable current source 92 to the signals V1 and V2 inputted to thepositive input of the differential amplification circuit 93 from thematrix circuit 72 and the signals V3 and V4 inputted to the negativeinput are converted from currents into voltages by the loads 91 a and 91b, respectively. The differential amplification circuit 93 of thecomparison section 62 executes arithmetic operation of the expression(7) given hereinabove from the signals V1 to V4 and the bias BIAS insynchronism with a comparison section drive pulse signal (not shown)from the horizontal scanning circuit 33 a (in FIG. 9, an arithmeticoperation phase). Then, a result of the arithmetic operation isoutputted to the outputting section 54. In the example of FIG. 9, “1” isoutputted, and this indicates that a peak of the received infrared raysintensity has been detected.

In step S18, the outputting section 54 outputs the result of thearithmetic operation as a pixel signal to the outputting circuit 34 overthe common signal line 42 (in FIG. 9, an output phase) in synchronismwith a selection signal from the horizontal scanning circuit 33 a (whichcorresponds to the sampling timing k+1 of the sampling synchronizingsignal).

In step S19, the outputting circuit 34 outputs the pixel signal to theshape data processing section 18 in synchronism with a control pulsesignal from the timing generator 32 and ends the processing.

It is to be noted that the processing described above is repeated foreach sampling timing as seen from FIG. 9. In particular, the processingenters an accumulation phase of the arithmetic operation mode A at areset pulse signal immediately after the sampling timing k−1, and entersan arithmetic operation phase at a clear pulse signal CLR4. Then, with areset pulse signal at the sampling timing k, the processing enters anext accumulation phase of an arithmetic operation of the mode B, andthereafter, the cycle described is repeated.

In the foregoing description, the light reception section 51,amplification section 52, arithmetic operation section 53 and outputtingsection 54 are provided in each of the pixels 41. However, thearithmetic operation section 53 and the outputting section 54 mayalternatively be provided outside the pixel 41 as seen in FIG. 10. FIG.10 shows a modified distance sensor 17 wherein the arithmetic operationsection 53 and the outputting section 54 of each of the pixels 41 of thedistance sensor 17 which corresponds to that of FIG. 2 are providedseparately.

In FIG. 10, like elements to those of FIG. 2 are denoted by likereference characters and overlapping description of them is suitablyomitted herein to avoid redundancy. In the distance sensor 17 of FIG.10, pixel output lines 101 and a storage arithmetic operation area 102are provided newly. Pixel signals from the pixels 41 in the optical area31 are outputted to the corresponding storage sections 61 in the storagearithmetic operation area 102 over the corresponding pixel output lines101. Each of the pixels 41 shown in FIG. 10 includes the light receptionsection 51 and the amplification section 52 of FIG. 4, and elementsserving as the arithmetic operation section 53 and the outputtingsection 54 following the light reception section 51 and theamplification section 52, respectively, are provided in the storagesection 61 corresponding to the pixel 41 of the storage operation area102. The arithmetic operation section 53 and the outputting section 54are connected to a corresponding one of the pixel output lines 101.Where the arithmetic operation sections and the outputting sections ofthe pixels 41 are provided separately in this manner, the pixels 41 canbe arranged efficiently in the optical area 31.

FIG. 11 is a timing chart illustrating processing of a plurality ofpixels 41 of the optical area 31 and the storage arithmetic operationarea 102 of FIG. 10. In particular, for example, with regard to thepixel i−1, the storage section 61 of the storage arithmetic operationarea 102 stores, at the sampling timings k and k+1 of FIG. 9, a signalinputted thereto in synchronism with a transfer pulse signal from thelight reception section 51 through the amplification section 52 and overthe pixel output line 101 into a storage cell (one of the storage cells71 a to 71 d in which data of the oldest timing is stored) from whichdata has been erased in synchronism with a clear pulse signal. Then, thelight reception section 51 corresponding to the storage section 61 isreset by a reset pulse signal and starts light reception newly. On theother hand, the signals stored in the storage cells 71 a to 71 d arearithmetically operated in response to an arithmetic operation selectionsignal inputted from the arithmetic operation control section 35, and aresult of the arithmetic operation is outputted to the outputtingsection 54. Then, the outputting section 54 outputs the received signalto the outputting circuit 34 over the common signal line 42 insynchronism with a selection signal from the horizontal scanning circuit33 a. Thereafter, the next pixel i executes the processing illustratedin FIG. 9 at the sampling timings k and k+1 same as those for the pixeli−1. It is to be noted that, while the distance sensor 17 of FIG. 10 isof the horizontal scanning type same as that of FIG. 2, it may have aconstruction otherwise of the vertical scanning type shown in FIG. 3.

While, in the foregoing description, the image processing apparatusexecutes three-dimensional image processing, it may execute some otherprocessing which requires arithmetic operation processing together withimage processing. For example, the image processing apparatus may beapplied also to a thermography apparatus which measures a temperaturedistribution together with image information, or may be applied to athree-dimensional thermography apparatus by combining thethree-dimensional image processing described above and the thermographyapparatus.

Where each of the pixels 41 has an arithmetic operation function asdescribed above, image processing on the real-time basis is allowed.

While preferred embodiments of the present invention have been describedusing specific terms, such description is for illustrative purposesonly, and it is to be understood that changes and variations may be madewithout departing from the spirit or scope of the following claims.

1. An image processing apparatus having an optical area in which aplurality of elements are disposed in a matrix having rows and columns,comprising: light reception means for receiving light introduced intosaid elements of said optical area and photoelectrically converting thelight; a plurality of operating units, each of which operates a signalobtained for a corresponding one of said elements by the photoelectricconversion by said light reception means in accordance with apredetermined rule, the operation of said operating units being based onat least one of a clear signal and a transfer signal; a plurality ofoutputting units, each for receiving a result of the operation of acorresponding one of said operating units and outputting the result ofthe operation for each of said corresponding one of said elements; andtiming adjustment means for adjusting a timing at which the result ofthe operation is output for each of said plurality of elements from saidoutputting units, said timing adjustment means using a control signalother than the clear signal or the transfer signal in the timingadjustment.
 2. An image processing apparatus according to claim 1,wherein said operating units include storage means for successivelystoring a plurality of signals at different timings obtained by thephotoelectric conversion.
 3. An image processing apparatus according toclaim 2, wherein said operating units execute a comparison operation fora combination of a plurality of ones of the signals stored in saidstorage means.
 4. An image processing apparatus according to claim 3,wherein the comparison operation includes an arithmetic operation fordetermining a maximum value or a minimum value of the signal.
 5. Animage processing apparatus according to claim 1, wherein said outputtingunits output results of an arithmetic operation for each of the rows orthe columns of said corresponding elements at a timing adjusted by saidtiming adjustment means.
 6. An image processing apparatus according toclaim 1, wherein said operating units perform an arithmetic operation.7. An image processing apparatus according to claim 1, wherein theoperating units store and operate a plurality of photoelectricallyconverted signals received by one of said elements at predeterminedtiming intervals.
 8. An image processing apparatus according to claim 7,wherein the operating units store and operate the photoelectricallyconverted signals in the order in which the signals are received.